To solve the traffic optimization problem of Application Layer Multicast (ALM) in Peer-to-Peer (P2P) system, a real-time flow scheduling system based on Software Defined Network (SDN) was designed. Firstly, some network measurement technologies were used to obtain the traffic matrix of the network, which was then abstracted into a weighted network state diagram and provided to the Terminal First Steiner Tree (TFST) generation algorithm. The TFST generation algorithm was divided into two stages. When generating multicast tree in the first stage, the algorithm was dexterously made to give higher priority to the terminal nodes by modifying the distance of terminal nodes to zero. In the second stage, the amount of branch nodes was adjusted according to the preset weighting factor so that the calculated multicast tree was able to give consideration to both traffic cost and implementation cost. Finally, to prevent the degradation of network performance caused by the too frequent updates of flow table when deploying the multicast tree into the network, a recurrent neural network-based module was designed to automatically adjust the update cycle according to the network performance. The simulation results indicate that the congestion index of the network using ALM real-time flow scheduling system is reduced by 47% compared with that of the original network. In addition, under medium load, the average value of the congestion index of the method with neural network module used to automatically adjust the update cycle is reduced by 17.6% and 25% respectively compared with those of the immediate update and fixed 5-second interval update methods. It is seen that the design has great practical application value in introducing machine learning into SDN to realize intelligent network.